{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "368686b4-f487-4dd4-aeff-37823976529d",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/multi_modal/azure_openai_multi_modal.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "# Multi-Modal LLM using Azure OpenAI GPT-4o mini for image reasoning\n",
    "\n",
    "In this notebook, we show how to use GPT-4o mini with the **Azure** OpenAI LLM class/abstraction for image understanding/reasoning. For a more complete example, please visit [this notebook](https://github.com/run-llama/llama_index/blob/main/docs/examples/multi_modal/openai_multi_modal.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d57f059",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-llms-azure-openai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1a5b0f05aadc911",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "1. Setup an Azure subscription - you can create one for free [here](https://azure.microsoft.com/en-us/free/cognitive-services/)\n",
    "2. Apply for access to Azure OpenAI Service [here](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUOFA5Qk1UWDRBMjg0WFhPMkIzTzhKQ1dWNyQlQCN0PWcu) \n",
    "3. Create a resource in the Azure portal [here](https://portal.azure.com/?microsoft_azure_marketplace_ItemHideKey=microsoft_openai_tip#create/Microsoft.CognitiveServicesOpenAI)\n",
    "4. Deploy a model in Azure OpenAI Studio [here](https://oai.azure.com/)\n",
    "\n",
    "\n",
    "You can find more details in [this guide.](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal)\n",
    "\n",
    "Note down the **\"model name\"** and **\"deployment name\"**, you'll need it when connecting to your LLM."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4479bf64",
   "metadata": {},
   "source": [
    "##  Use GPT-4o mini to understand Images from URLs / base64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5455d8c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"AZURE_OPENAI_API_KEY\"] = \"xxx\"\n",
    "os.environ[\"AZURE_OPENAI_ENDPOINT\"] = \"https://YOUR_URL.openai.azure.com/\"\n",
    "os.environ[\"OPENAI_API_VERSION\"] = \"2024-02-15-preview\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d0d083e",
   "metadata": {},
   "source": [
    "## Initialize `AzureOpenAI` and Load Images from URLs\n",
    "\n",
    "Unlike regular `OpenAI`, you need to pass the `engine` argument in addition to `model`. The `engine` is the name you \n",
    "gave to your model when you deployed it in Azure OpenAI Studio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e5c6b095ed0995d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.azure_openai import AzureOpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf65789c9a66cc8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "azure_openai_llm = AzureOpenAI(\n",
    "    engine=\"my-gpt-4o-mini\",\n",
    "    model=\"gpt-4o-mini\",\n",
    "    max_new_tokens=300,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa22796cdd358274",
   "metadata": {},
   "source": [
    "Alternatively, you can also skip setting environment variables, and pass the parameters in directly via constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d310e814a6da1c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "azure_openai_llm = AzureOpenAI(\n",
    "    azure_endpoint=\"https://YOUR_URL.openai.azure.com/\",\n",
    "    engine=\"my-gpt-4o-mini\",\n",
    "    api_version=\"2024-02-15-preview\",\n",
    "    model=\"gpt-4o-mini\",\n",
    "    max_new_tokens=300,\n",
    "    api_key=\"xxx\",\n",
    "    supports_content_blocks=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8725b6d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import requests\n",
    "from llama_index.core.schema import Document, MediaResource\n",
    "\n",
    "image_url = \"https://www.visualcapitalist.com/wp-content/uploads/2023/10/US_Mortgage_Rate_Surge-Sept-11-1.jpg\"\n",
    "\n",
    "response = requests.get(image_url)\n",
    "if response.status_code != 200:\n",
    "    raise ValueError(\"Error: Could not retrieve image from URL.\")\n",
    "img_data = base64.b64encode(response.content)\n",
    "\n",
    "image_document = Document(image_resource=MediaResource(data=img_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05d94bcb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img width=400 src=\"\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "src = f'<img width=400 src=\"data:{image_document.image_resource.mimetype};base64,{image_document.image_resource.data.decode(\"utf-8\")}\"/>'\n",
    "HTML(src)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbd9c116",
   "metadata": {},
   "source": [
    "### Complete a prompt with an image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c96ab53e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.llms import (\n",
    "    ChatMessage,\n",
    "    ImageBlock,\n",
    "    TextBlock,\n",
    "    MessageRole,\n",
    ")\n",
    "\n",
    "msg = ChatMessage(\n",
    "    role=MessageRole.USER,\n",
    "    blocks=[\n",
    "        TextBlock(text=\"Describe the images as an alternative text\"),\n",
    "        ImageBlock(image=image_document.image_resource.data),\n",
    "    ],\n",
    ")\n",
    "\n",
    "response = azure_openai_llm.chat(messages=[msg])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3eba4477",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assistant: The image presents a graph titled \"The U.S. Mortgage Rate Surge,\" comparing the U.S. 30-year fixed-rate mortgage rates with existing home sales from 2014 to 2023. \n",
      "\n",
      "- The vertical axis on the left represents the mortgage rate, while the right vertical axis indicates the number of existing home sales, measured in millions.\n",
      "- A blue line illustrates the trend of existing home sales, showing fluctuations over the years, peaking around 2020 and declining thereafter.\n",
      "- A red line represents the mortgage rate, which has seen a significant increase, particularly in 2022 and 2023, reaching its highest level in over 20 years.\n",
      "- The background includes a subtle grid, and the data sources are noted at the bottom. The overall design is clean and informative, aimed at highlighting the relationship between rising mortgage rates and declining home sales.\n"
     ]
    }
   ],
   "source": [
    "print(response)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama-index-llms-azure-openai-PD0tIpEp-py3.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
